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Abstract 


Massive open online courses (MOOCs) are a recent addition to the range of online 
learning options. Since 2008, MOOCs have been run by a variety of public and elite 
universities, especially in North America. Many academics have taken interest in 
MOOCs recognising the potential to deliver education around the globe on an 
unprecedented scale; some of these academics are taking a research-oriented 
perspective and academic papers describing their research are starting to appear in the 
traditional media of peer reviewed publications. This paper presents a systematic review 
of the published MOOC literature (2008-2012): Forty-five peer reviewed papers are 
identified through journals, database searches, searching the Web, and chaining from 
known sources to form the base for this review. We believe this is the first effort to 
systematically review literature relating to MOOCs, a fairly recent but massively popular 
phenomenon with a global reach. The review categorises the literature into eight 
different areas of interest, introductory, concept, case studies, educational theory, 
technology, participant focussed, provider focussed, and other, while also providing 
quantitative analysis of publications according to publication type, year of publication, 
and contributors. Future research directions guided by gaps in the literature are 
explored. 

Keywords: MOOC; massive open online course; massively open online course; 
systematic review; connectivism 
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Introduction 


Distance education has a long history, with correspondence courses making use of 
reasonable cost universal postal services for the delivery of study material to learners 
and for submission/return of assignments by/to students (Casey, 2008). Further 
developments of distance education have appeared with each new communication 
technology: radio, television, video recorders, home computing. The latest development, 
that of the Internet (including very recently the mobile Internet), has similarly been 
adopted by many existing higher education providers but has also supported the 
emergence of a new model dubbed a massive open online course (MOOC(s)), the term 
coined in 2008 to describe an open online course to be offered by the University of 
Manitoba in Canada. A range of both topics and platforms have since emerged and the 
term was described as “the educational buzzword of 2012” by Daniel (2012) reflecting 
widespread interest in the concept. MOOCs are widely discussed across a range of 
media, including blogs and the specialist and popular press, however this includes 
“thinly disguised promotional material by commercial interests ... and articles by 
practitioners whose perspective is their own MOOC courses” according to Daniel (2012). 

This paper seeks to classify academic research relating to MOOCs, based on a systematic 
review of the existing peer reviewed MOOC literature. Search techniques for papers 
related to MOOCs are considered and a corpus of papers identified, then a grounded 
research approach is presented from which a classification of the works emerges. 


Background 


Since the early days of computing, academics have shared digital content (Lane & 
McAndrew, 2010) and recently there has been much interest in the sharing of open 
educational resources (OER), particularly relating to higher education, which has also 
become an important resource base for teachers and learners (Adams, 
Liyanagunawardena, Rassool, & Williams, 2013). In 2001 the Massachusetts Institute of 
Technology (MIT) launched its pioneering OpenCourseWare (OCW), with the aim of 
publishing materials from all its courses permanently on the open Web, with licenses 
allowing its use, modification, and redistribution. Since then many other established 
universities have joined the movement such as the Open University of UK through the 
OpenLearn project and the Open Learning Initiative by Carnegie Mellon University. 
OER were made available for two purposes: Learners could access the material directly 
and, hopefully, learn from it; educators could use the material as part of their own 
teaching (as produced or by amending it themselves). A significant proportion of these 
OER, however, were of limited use since they were usually produced in order to be a 
specific part of a larger educational experience within a specific educational framework. 
This limitation was particularly frustrating for many aspiring learners attempting to use 
them directly, but could also cause problems when used naively by educators 
(Liyanagunawardena, 2012; Weller, 2007). The concept of open access to learning was 
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taken in a different direction with the introduction of the massive open online courses 
or MOOCs (Fini, 2009). 

A MOOC brings together people interested in learning (or “students”) and an expert or 
experts who seek to facilitate the learning. Connectivity is usually provided through 
social networking, and a set of freely accessible online resources provides the content or 
the study material. Furthermore, they generally have no prerequisites, fees, formal 
accreditation, or predefined required level of participation (McAuley, Stewart, Siemens, 
& Cormier, 2010). Participation in a MOOC is completely voluntary and is dependent on 
the interested individual. The collaborative space of a MOOC can span across many 
different platforms and technologies. For example, MOOC participants may create their 
own blog posts discussing aspects of the MOOC in different spaces and/or may use 
microblogs such as Twitter to express themselves. “Connectivism and Connective 
Knowledge”, an online course offered through the Learning Technologies Centre and 
Extended Education at the University of Manitoba and facilitated by George Siemens 
and Stephen Downes (Downes, 2008), is considered the first MOOC. This online course 
had 25 paid enrolments (for credit) with around 2,200 non-credit, non-fee paying 
students. It used the principles of connectivism (Siemens, 2005) and unlike the 
traditional form of online learning was not primarily relying on resources posted 
through a learning management system (LMS)'. 

More recently MOOCs have developed within international co-operative partnerships 
such as Coursera (www.coursera.org) . a partnership of 62 world class universities (as of 
April 11, 2013) led by Stanford University, and edX (www.edx.org) which includes the 
Massachusetts Institute of Technology, Ecole Polytechnique Federale de Lausanne, The 
Hong Kong University of Science and Technology. Udacity (www.udacitv.com) . P2P 
University, and Futurelearn (the UK Open University’s MOOC platform) are other 
related platforms. The numbers registering for MOOCs have reached 160,000 in the 
case of a 2011 Artificial Intelligence online course offered by Stanford University 
(Rodriguez, 2012). 

With the increasing uptake and interest in MOOCs, it has become a popular topic in the 
educational press such as the Time Higher Education Magazine article (Corbyn, 2012), 
which presents an account of running a MOOC. Likewise there are many blog posts 
relating to MOOCs, posted from a variety of viewpoints including course leaders, 
participants, and outsiders (for example http://mooctalk.org chronicles the experience 
of a mathematician leading a MOOC in late 2012). 

Academic papers on MOOCs began to appear in the peer reviewed literature (such as 
journals, conference proceedings, and professional magazines) in 2008 (Downes, 
2008), with an increasing number of papers appearing each year since. This 
phenomenon is similar to many other technologies that created interest in academic 
communities. For example, Twitter appeared as a micro-blogging tool in 2006 and the 
literature on Twitter has grown exponentially from three papers in 2007 to hundreds of 
papers in 2011 (Williams, Terras, & Warwick, 2013). 
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Method 


Data Collection 

Researchers use different methods to identify papers to consider for a literature review 
(Ellis 1989; Ellis & Haugan, 1997): Methods include searching in databases or search 
engines and chaining from known research papers. For a systematic review it is 
important that the methods of identifying papers are described and justified, and that 
the approach can be re-applied by others (Fink, 2010). The aim of this study was to 
locate and analyse MOOC related academic literature to provide an understanding of 
developing research areas, methods applied in research, and topics lacking published 
research. 

Relevant papers were identified through a series of search efforts, using an approach 
based on the methods used in other systematic reviews including two studies of 
literature related to the microblogging system Twitter (Gao, Luo, & Zhang, 2012; 
Williams, Terras, & Warwick, 2013). Papers were classed as relevant if their primary 
focus was to explore the concept of a MOOC or the implications for higher education, 
report on experiments with MOOCs, or compare MOOCs with other educational 
approaches. There was insufficient statistical data across the papers found to undertake 
a meta-analysis (Fink 2010). 

Firstly, the search terms and boundaries to be used were established. Initially two 
search terms (and their plurals) were selected: 

• MOOC 

• Massively Open Online Course 

However it was identified that some authors used “Massive” instead of “Massively” (for 
example Kop, Fournier, & Mak 2011) and so the third term (and its plural) was added: 

• Massive Open Online Course 

The search period was limited to the period from the year in which the first MOOC was 
run (2008) to the year this study started (2012). Where possible the search was limited 
to titles and abstracts to reduce the workload of manual filtering of irrelevant papers. 

Secondly, following the approach of Gao, et al. ( 2012), a number of academic journals 
in the disciplines of educational technology and distance education were selected for a 
preliminary search (Table 1). These journals were The British Journal of Educational 
Technology, Distance Education, American Journal of Distance Education, and 
Journal of Online Learning and Teaching. The search returned a small number of 
papers in one of these journals, but on detailed examination none of these were relevant 
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for this systematic review. For example Roderick (2008) was predominantly about 
connected learning in digital spaces, and massively multiplayer online role-playing 
games were discussed as an example of connectivity. We believe that even though the 
search terms were used within inverted commas as phrases, the search algorithm may 
have returned articles such as this by comparing single search terms. 

Table 1 

Search Results - Selected Academic Journals 


Journal name 

Search results 

Relevant 

Date 

British Journal of Educational Technology 

0 

0 

10/12/2012 

Distance Education 

16 

0 

10/01/2013 

American Journal of Distance Education 

0 

0 

30/11/2012 

Journal of Online Learning and Teaching 

0 

0 

30/11/2012 


Thirdly, the same search terms were then used to search various academic databases: 
ISI Web of Knowledge, ProQuest (ERIC, British Education Index and Australian 
Education Index), JSTOR (education titles), IEEEXplorer, and Scopus (Table 2). A 
number of the articles returned were unrelated to this study, for example papers 
returned included some on “Multiple Optical Orthogonal Code Sequences” and some on 
“Management of Organizational Change” 2 , both of which were abbreviated to MOOC. 

Table 2 

Search Results - Selected Academic Databases 


Database 

Search results 

Relevant 

Date 

ISI Web of Knowledge 

5 

2 

17/11/2012 

ProQuest 

6 

6 

30/11/2012 

JSTOR (education titles) 

95 

0 

30/11/2012 

IEEEXplorer 

1 

1 

30/11/2012 

Scopus 

39 

12 

04/12/2012 
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Fourthly, the same search was conducted using the Google Scholar search engine with 
94 results (November 21, 2012). From these results only 33 were found to be relevant. 
However, the same search repeated on Google Scholar later (on January 08, 2013) only 
resulted in 25 results suggesting that some articles previously found were not online 
and/or Google had changed its search algorithms. For example, the first result returned 
by the search done on the 21 st November 2012 was the article by Fini (2009); but the 
recent search that returned only 25 results did not contain this article. Presentations 
and papers that did not discuss MOOCs as the primary concern of the paper were 
discounted after a careful reading of the abstract. 

In order to improve the coverage of relevant publications to be included in this review, 
the chaining technique of Gao, et al. (2012) was used: consulting the reference lists of 
papers that were already in the corpus to locate other relevant work. This resulted in the 
addition of one more article to the corpus. Google Scholar searches were also performed 
for each of the articles in the corpus to identity other articles that had referenced them 
(forward referencing). This resulted in the addition of one more conference paper. 
During December 2012 two additional papers were identified: one in the 
Communications of the ACM and another in the Journal of Interactive Media in 
Education. 

The data collection process resulted in the identification of 45 distinct articles - 17 from 
journals, 13 conference publications (including one poster conference presentation), 10 
academic magazine articles, 3 reports, and 2 workshop presentations. 

Data Classification/Analysis 

Articles were classified both quantitatively and qualitatively. The quantitative analysis 
was used to classify the papers according to the publication year and the type of 
publication in which the article appeared. Papers were qualitatively classified using 
open coded content analysis, a technique used by the two studies of literature relating to 
Twitter (Gao, et al., 2012; Williams, Terras, & Warwick 2013). Initially the first author 
read each of the papers to identify themes, types of inquiry employed, and the future 
research directions indicated. These classifications were refined in an iterative manner 
by all authors reading and re-reading articles, and considering alternative classifications 
and stratifications. 
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Results 


Quantitative Details 

The first MOOC related paper was published in 2008, with again just one paper 
identified in 2009, seven papers in 2010, 10 in 2011, and 26 in 2012. Clearly an 
increasing pattern in the number of articles published on MOOCs can be seen from 
Figure 1. 


MOOCs Articles by Publication 
Year 



Publication year 


Figure 1. Articles by publication year. 


The majority of identified articles published to date are in journals (17 papers), with a 
smaller number of articles appearing in conference proceedings and magazines (13 and 
10 respectively). Figure 2 diagrammatically presents the percentage of papers in each 
type of publication. 
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Figure 2. Article classification by publication type. 


Figures illustrates the separation of articles by publication type and year. From 2009 to 
2012 there is a gradual increase of the number of journal articles and conference papers. 
It is noticeable that the first magazine article about MOOCs appeared in 2011 and the 
following year there is a fourfold increase, reflecting the recent general interest in 
MOOCs. 
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Articles by Type and Publication Year 



■ Journal 

■ Conference 

■ Magazine 

■ Workshop 

■ Reports 


Publication Year 


Figure 3. Articles by publication year and type. 


It is noticeable that while many of the publications identified only had one article on 
MOOCs, exceptionally among the journals the International Review of Research in 
Open and Distance Learning published six articles on MOOCs, while the European 
Journal of Open Distance and E-Learning published three. Communications of the 
ACM published the highest number of magazine articles (three) while eLearn Magazine 
and Learning Solutions magazine published two articles each. The International 
Conference on Networked Learning published three articles on MOOCs, two in 2010 
and another in 2011. 

A list of authors with more than one article being reviewed here are presented in Table 
3. Kop has the highest number of publications (five) in the review while deWaard, 
Hogue, Koutropoulos, and Rodriguez have four each. It can be seen that the authors 
with more than three publications have worked in teams and several such teams can be 
identified from co-authorship, for example Kop and Fournier; de Waard, Abajian, 
Gallagher, Hogue, Keskin, Koutropoulos and Rodriguez; Mackness, Mak, and Williams. 

Table 3 


Contributors with Multiple Publications in the Review 


Author 

No. of 
articles 

Details 

Kop 

5 

Kop & Fournier 2010; Kop 2011; Fournier, Kop & Sitlia 2011; 
Kop, Fournier & Mak 2011; Kop & Carroll 2012 

deWaard 

4 

deWaard 2011; 

deWaard, Abajian, Gallagher, Hogue, Keskin, Koutropoulos & 
Rodriguez 2011; 


|Vol 14 | N03 


July/13 


210 













MOOCS : A Systematic Study of the Published Literature 2008-2012 


Liyanagunawardena, Adams, and Williams 




deWaard, Koutropoulos, Keskin, Abajian, Hogue, Rodriguez 
& Gallagher 2011; 

Koutropoulos, Gallagher, Abajian, deWaard, Hogue, Keskin, 
& Rodriguez 2012 

Hogue 

4 

deWaard, Abajian, Gallagher, Hogue, Keskin, Koutropoulos & 
Rodriguez 2011; 

deWaard, Koutropoulos, Keskin, Abajian, Hogue, Rodriguez 
& Gallagher 2011; 

Koutropoulos, Gallagher, Abajian, deWaard, Hogue, Keskin & 
Rodriguez 2012; 

Koutropoulos, & Hogue 2012 

Koutropoulos 

4 

deWaard, Abajian, Gallagher, Hogue, Keskin, Koutropoulos & 
Rodriguez 2011; 

deWaard, Koutropoulos, Keskin, Abajian, Hogue, Rodriguez 
& Gallagher 2011; 

Koutropoulos, Gallagher, Abajian, deWaard, Hogue, Keskin & 
Rodriguez 2012; 

Koutropoulos, & Hogue 2012 

Rodriguez 

4 

deWaard, Abajian, Gallagher, Hogue, Keskin, Koutropoulos & 
Rodriguez 2011; 

deWaard, Koutropoulos, Keskin, Abajian, Hogue, Rodriguez 
& Gallagher 2011; 

Koutropoulos, Gallagher, Abajian, deWaard, Hogue, Keskin & 
Rodriguez 2012; 

Rodriguez 2012; 

Abajian 

3 

deWaard, Abajian, Gallagher, Hogue, Keskin, Koutropoulos & 
Rodriguez 2011; 

deWaard, Koutropoulos, Keskin, Abajian, Hogue, Rodriguez 
& Gallagher 2011; 

Koutropoulos, Gallagher, Abajian, deWaard, Hogue, Keskin & 
Rodriguez 2012 

Fournier 

3 

Kop & Fournier 2010; Fournier, Kop & Sitlia 2011; 

Kop, Fournier & Mak 2011; 

Gallagher 

3 

deWaard, Abajian, Gallagher, Hogue, Keskin, Koutropoulos & 
Rodriguez 2011; 

deWaard, Koutropoulos, Keskin, Abajian, Hogue, Rodriguez 
& Gallagher 2011; 

Koutropoulos, Gallagher, Abajian, deWaard, Hogue, Keskin & 
Rodriguez 2012 

Keskin 

3 

deWaard, Abajian, Gallagher, Hogue, Keskin, Koutropoulos & 
Rodriguez 2011; 

deWaard, Koutropoulos, Keskin, Abajian, Hogue, Rodriguez 
& Gallagher 2011; 

Koutropoulos, Gallagher, Abajian, deWaard, Hogue, Keskin & 
Rodriguez 2012 

Mak 

3 

Mackness, Mak & Williams 2010; Mak, Williams & Mackness 
2010; 

Kop, Fournier & Mak 2011 

Mackness 

3 

Mackness, Mak & Williams 2010; Mak, Williams & Mackness 
2010; 

Tschofen & Mackness 2012 

Bell 

2 

Bell (2010a); Bell (2010b) 

Levy 

2 

Levy 2011; Schrire & Levy 2012 

Williams 

2 

Mackness, Mak & Williams 2010; Mak, Williams & Mackness 
2010 
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Publication Topics/Themes 

The articles embraced a wide-range of themes relating to MOOCs; an initial list 
included 1 : 

• agency 

• connectivism 

• actor network theory 

• dangers 

• learner experience 

• pedagogies 

• technology 

• trends 

Most papers included some introduction to the term MOOC and a history of the 
concept. The majority of articles were primarily concerned with the concept of MOOCs, 
discussing challenges and trends, while other themes generally appeared within only 
one paper except for the concept of connectivism and its implications. The themes were 
re-stratified into the following categories. 

1. Introductory: explaining aspects of MOOCs. 

2. Concept: encompassing discussion papers on topics such as the threats and 
opportunities of MOOCs for Higher Education and its existing institutions. 

3. Case studies: examining one or more MOOCs (including papers studying the 
same course running in different years and papers studying different courses). 

4. Educational theory: considering the pedagogic approaches used. 

5. Technology: presenting details or consideration of the software and hardware 
used. 

6. Participant focussed: considering aspects related to the learners participating in 
MOOCs. 

7. Provider focussed: considering aspects related to the provider of the MOOC, 
including the course creators and leaders. 

8. Other: this category was introduced to cover the two articles that did not come 
under any other category. These were Esposito (2012), which discussed ethical 
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issues in using data generated through a MOOC, and Frank (2012), which 
presented the author’s views on alternative ways to run MOOCs. 

The articles were then individually re-considered and assigned to the categories which 
reflected the contents; many articles were assigned to more than one category. Table 4 
shows the spread of these categories. 

Table 4 

Article Categorization 


Category 

Articles 

Introductory 

McAuley, Stewart, Siemens & Cormier 2010; Koutropoulos & 

Hogue 2012; Rodriguez 2012; Bremer 2012; deWaard 2011; Kop & 
Carroll 2012; Masters 2011; Roberts 2012; Mahraj 2012; Daniel 
2012; Hyman, 2012 

Concept 

Martin 2012; Bull 2012; Kirkwood 2010; Ardis & Henderson 2012; 
Bonino 2012; Vardi 2012; Mehaffy 2012; Anderson & McGreal 

2012; Butin 2012; Mahraj 2012; Mehlenbacher 2012; Anderson & 
McGreal 2012; Hyman, 2012 

Case Studies 

Bell 2010a; Bell 2010b; Bremer 2012; deWaard, Abajian, 

Gallagher, Hogue, Keskin, Koutropoulos & Rodriguez 2011; 
deWaard,Koutropoulos, Keskin, Abajian, Hogue, Rodriguez & 
Gallagher 2011; Downes 2008; Fini 2009; Fournier, Kop & Sitlia 
2011; Kop & Fournier 2010; Kop 2011; Kop, Fournier & Mak 2011; 
Kop & Carroll 2012; Koutropoulos, Gallagher, Abajian, deWaard, 
Hogue, Keskin, & Rodriguez 2012; Levy 2011; Mackness, Mak & 
Williams 2010; Mak, Williams & Mackness 2010; Roberts 2012; 
Rodriguez 2012; Stewart 2010; Schrire & Levy 2012; Vihavainen, 
Luukkainen & Kurhila 2012 

Educational 

Theory 

Bell 2010a; Bell 2010b; Butin 2012; Cabiria 2012; deWaard, 

Abajian, Gallagher, Hogue, Keskin, Koutropoulos & Rodriguez 

2011; deWaard,Koutropoulos, Keskin, Abajian, Hogue, Rodriguez 
& Gallagher 2011; Downes 2008; Kop & Fournier 2010; Kop, 
Fournier & Mak 2011; Mackness, Mak & Williams 2010; Mak, 
Williams & Mackness 2010; McAuley, Stewart, Siemens & Cormier 
2010; Rodriguez 2012; Stewart 2010; Tschofen & Mackness 2012 

Technology 

Anderson & McGreal 2012; Fini 2009; Kop, Fournier & Mak 2011; 
Kop & Carroll 2012; Mak, Williams & Mackness 2010; McAuley, 
Stewart, Siemens & Cormier 2010; Rodriguez 2012; Vihavainen, 
Luukkainen & Kurhila 2012 

Participant 

focussed 

Chamberlin & Parish 2011; Kop & Fournier 2010; Kop 2011; Kop & 
Carroll 2012; Koutropoulos, Gallagher, Abajian, deWaard, Hogue, 
Keskin, & Rodriguez 2012; Levy 2011; Mackness, Mak & Williams 
2010; Mak, Williams & Mackness 2010; Stewart 2010 

Provider focussed 

Maclsaac 2012; Mahraj 2012; Sadigh, Seshia & Gupta 2012 

Other 

Esposito (2012); Frank (2012) 
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Case Studies 

Across the articles classified, 21 had a case study element, and in total 13 different 
MOOCs were studied. 


• 

CCK08 - Connectivism and Connective Knowledge: the first MOOC offered 
from the University of Manitoba (Canada) in 2008. 

• 

CCK09 - Connectivism and Connective Knowledge: a later version of 
CCK08 offered in 2009. 

• 

CCK11 - Connectivism and Connective Knowledge: a later version of CCK08 
offered in 2011. 

• 

CritLit - Critical Literacies: a 2010 course blending 'critical thinking' and 
'21st century literacies' offered by the National Research Council of Canada. 

• 

CSi - An introductory programming course from the University of Helsinki 
(Finland) based on an apprenticeship model. 

• 

CS101 - An introductory computer science course, from Udacity. 

• 

CS221 - A course on artificial intelligence offered by Stanford (US). 

• 

EduMOOC - Online Learning Today... and Tomorrow: offered by the 
University of Illinois (US) in 2011. 

• 

FSLT11 - First Steps in Learning and Teaching, a 2011 course offered by 
Oxford Brookes University (UK). 

• 

Future of Learning - an education course: offered by the University of 
Frankfurt (Germany), in 2011. 

• 

MobiMOOC - Mobile learning: offered by the Institute of Tropical Medicine 
Antwerpen (Belgium) in 2011. 

• 

PLENK2010 - Personal Learning Environments, Networks, and Knowledge: 
a 2010 course offered by Athabasca University (Canada). 

• 

Qual MOOC - Qualitative Research Methodology: Kibbutzim College of 
Education, Technology and the Arts, Israel proposed to be run in 2012. 


The MOOCs CCK08, CCK11, CritLit, and PLENK2010 have designers and deliverers in 
common. Of the case study papers, one (Rodriguez, 2012) compared different MOOCs 
(including CCK08, PLENK2010, MobiMOOC, EduMOOC, Stanford CS221, and CS101 
from Udacity); two (Kop 2011; Kop, Fournier, & Mak, 2011) considered PLENK2010 and 
one other MOOC (CritLit and CCK11, respectively); and another (Bell 2010a) considered 
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CCK08 and CCK09; the remainder focussed on single MOOCs. Table 5 shows the 
number of articles studying each MOOC and the number of articles studying each as the 
single central case study. The related MOOCs (PLENK2010 and CCK08) were the most 
studied. MobiMOOC is the only other course to be considered in more than one paper. 

Table 5 

Articles with MOOCs as Case Studies 


Name of the MOOC 

Total no. of 
articles 

No. of articles as 
single central case 

PLENK2010 

8 

5 

CCK08 

7 

5 

CCK09 

1 

0 

CCK11 

1 

0 

MobiMOOC 

4 

3 

Future of Learning 

1 

1 

CSi 

1 

1 

Qual MOOC 

1 

1 

EduMOOC 

1 

0 

CritLit 

1 

0 

CS101 

1 

0 

CS221 

1 

0 

FSLT11 

1 

1 


The majority of case studies have used multiple methods for data collection in line with 
the general practice of case study research (Yin, 2003). In most case studies online 
surveys were used to collect data from participants in the MOOC(s). SurveyMonkey 
( www.survevmonkev.com l and LimeSurvey ( www.limesurvev.org l were popular tools 
used in survey data collection. Four studies considering PLENK2010 as a case used 
multiple surveys (active participant survey, lurker survey, and end course survey). 
Researchers also reported collecting data via email interviews, focus groups, Moodle log 
data, discussion forum data, blogs, and observations. Three articles reported the use of 
techniques based on virtual ethnography and another used narrative inquiry as their 
research approach. 

Researchers’ role. 

Seven of the case study articles defined the researcher’s role in relation to the study. In 
two of these instances the researcher was described as a learner in the MOOC, in 
another two as a participant, and in the remaining three as an observer. The 
demarcation (if one really exists) between a learner and a participant in a MOOC is not 
clear, and may well simply be a matter of differing nomenclature. 
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Participants. 

The majority of the participants in the case study research were people who had 
participated in the MOOC under consideration. Two articles also included data collected 
from instructors. 

Classes of MOOC 

Rodriguez (2012) classifies MOOCs into two categories: connectivist MOOCs (c- 
MOOCs) and Al-Stanford like courses. He associates courses similar to Al-Stanford 
predominantly with cognitive-behaviourist approaches and c-MOOCs with connectivist 
approaches. By comparing multiple case studies Rodriguez (2012) claims that courses 
similar to Al-Stanford have a more individualist learning approach while c-MOOCs have 
a more social approach to learning. He also shows the different roles played by 
facilitators in each type of MOOC. Daniel (2012) on the other hand discusses cMOOCs 
and xMOOCs, which he claims are a bifurcation of MOOCs. He refers to “xMOOCs now 
being developed by elite US institutions that follow a more behaviourist approach” 
(Daniel 2012) but does not provide a definition. It is probable that both Rodriguez 
(2012) and Daniel (2012) are similarly classifying MOOCs but using the two different 
labels for the same thing: “Al-Stanford like courses” and “xMOOCs”. However, our 
systematic review has shown that there is a gap in the scholarly literature in defining 
these different types of MOOCs. 


Limitations 


The aim of this study is to develop a classification of academic research relating to 
MOOCs, based on a systematic consideration of the existing peer reviewed MOOC 
literature. Other materials such as blog posts and unpublished reports are not included 
in this review. Blog posts always present difficulties for authors of reviews in 
determining the credibility of the posts: There are few studies on how researchers use 
blogs (Shema, Bar-Ilan, & Thelwall, 2012); the transient nature of blogs means that they 
are difficult to suitably include in a systematic consideration. It should be noted that the 
same search terms applied above used in Google Web search returned over 50,000 
items and used in a Google Blog search resulted in 570 results (December 04, 2012), 
however these items are not analysed here. Daniel (2012) includes references to such 
resources. Inclusion of such resources in a systematic review is contentious as they are 
often highly subjective and have rarely been critically reviewed by peers. The question of 
whether such resources should be included in a systematic review needs further 
methodological consideration in the literature, but at present it is unusual due to issues 
of credibility. 

Articles published in languages other than English were not considered for this review. 
When searching forward references using Google Scholar, the authors came across 
publications that referenced articles in the review but were published in other languages 
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such as Chinese. Because of this, the authors believe that there could be articles that 
would have been included in the review were they published in English. 


Discussion 


This analysis seems to reflect the growing interest in MOOCs. However, most research 
has investigated the learner perspective, with a significant minor focus on the 
institutional threats and opportunities. The lack of published research on MOOC 
facilitators’ experience and practices leaves a significant gap in the literature. Mak, et al. 
(2010) suggest that there has been unacceptable behaviour (for example, forceful 
intellectual debates, feelings of participation being demanded, and rude behaviour) 
from some MOOC participants, which has led other participants to cease posting on 
forums. The possible cultural differences of participants in MOOCs and their MOOC 
experience would be an interesting avenue of research in relation to cultural tension in 
MOOCs. Many studies that used data generated from MOOCs have limited their 
qualitative analysis to postings on a formal LMS and a sample of other sources (such as 
blog posts) due to the volume of data (Kop, 2011; Kop, & Fournier, 2010). Inevitably this 
leaves out potentially useful data on Twitter, blogs, and other social media spaces. For 
example, Liyanagunawardena (2012) shows that there is a significant use of external 
communications (other than the official LMS discussions) to support learner groups in 
traditional, blended, and online distance education settings. 

Despite many research studies using publicly available data from a MOOC for research 
purposes, only a few papers have considered the ethical aspects of such use. This may 
well be due to a lack of good guidance and best practice examples on the general 
question of the use of data from online social media and similar sources for such 
research (Zimmer, 2010). This opens up a new avenue of research leading to MOOC 
participants’ (both students’ and facilitators’) views on the ethical issues of using the 
data generated by them on a MOOC for research and a researcher’s role in the MOOC. 
However this topic is discussed by MOOC participants and observers in blogs such as 
Robbins (2013). 

Many studies have presented participant demographics and it was observed that a large 
majority of participants were from North America and Europe. There were very few 
participants, if any at all, from Asia and Africa with a few from South East Asia 
(deWaard, Abajian, Gallagher, et ah, 2011; Kop, 2011; Koutropoulos, et ah, 2012). This is 
hardly surprising due to the possible barriers of access to online learning that can be 
presumed in those parts of the world, both technological and linguistic (see 
Liyanagunawardena, 2012 for a discussion of the similar problems in the introduction of 
online learning in Sri Lanka). However, a number of Asian and African learners 
participated in MobiMOOC (Koutropoulos, et ah 2012). There are two plausible reasons: 
It could be that the course content of the MobiMOOC (mLearning) is of more interest to 
learners from those regions or it could be that mLearning is significantly more 
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accessible in those places. deWaard, Koutropoulos et al. (2011) report that 77.5% of the 
participants in their survey had used mobile devices to access the course even though it 
was not required, indicating that the flexibility offered by mLearning is attractive for 
those MOOC participants. This shows the potential for exploring the possibility of 
mLearning MOOC provision in the developing world, both for increasing and widening 
participation. 

As discussed already, some of the researchers who used data generated through MOOCs 
have limited their qualitative analysis due to the large volumes. Regarding MOOC 
participant approaches, Fini (2009) similarly argues that 

people seem to be torn between the time-saving 
advantage offered by the ‘Daily’ solution [an email 
message sent daily by the facilitators with a summary of 
the topics of the existing conversation] and the multi¬ 
faceted, time-consuming alternative represented by 
direct access to unfiltered information. 

Due to the large volumes of data generated by the ‘massive’ number of students engaged 
in a MOOC, being up-to-date with the ongoing discussions can be challenging or even 
overwhelming. Exploring the strategies used by students who continue to be active 
participants in a MOOC could provide some insight into possible solutions to the 
information overload in a MOOC environment for both other MOOC learners and for 
MOOC researchers. In fact, Milligan, Margaryan, and Littlejohn (2013) show that all but 
one active participants of Changen MOOC have participated in previous MOOCs and 
question whether a learner has to learn how to learn in a MOOC. While many MOOC 
participants struggle to keep going “there’s some people who are everywhere you turn in 
the Changen MOOC: there’s this group of people who are inspirational, just 
phenomenal the way they just keep going and they know their way around it” (Milligan, 
Margaryan, & Littlejohn 2013). Confidence gained after successful participation in one 
MOOC together with network effects where people start building communities and 
networks in one MOOC then moving together onto another MOOC may result in more 
previous MOOC participants taking up other MOOCs. 

It is acknowledged that MOOCs have high withdraw/dropout rates (Koutropoulos, et al., 
2012); however, data on completion rates of MOOCs are not readily available. According 
to Jordan’s (2013) collated completion rates for 24 MOOCs (as of March 11th, 2013), the 
highest completion rate achieved was 19.2% on Functional Programming Principles in 
Scala, a MOOC offered by Coursera in 2012. The majority of MOOCs had completion 
rates of less than 10%. There is very little known about the experiences of non¬ 
completing MOOC participants (Koutropoulos, et al., 2012). This would be an 
interesting and useful avenue to explore further in future research, though engaging 
those who have started but not finished an educational course in research into the 
reasons for their non-completion is likely to be difficult. 
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Motivation is identified as an important contributor to student engagement in a MOOC 
by Milligan, Margaryan, and Littlejohn (2013). One can speculate about an individual’s 
motivation to participate in a MOOC: the desire to achieve an academic credential at a 
reduced cost, personal enrichment, and/or self satisfaction. However, why individuals 
participate in MOOCs has yet to be explored. It would be valuable to learn about the 
actual motivations in place, the percentage of participants taking up MOOCs for those 
reasons, and to know how those motivations might vary from one course or discipline or 
even provider to another. 

Recognition for MOOCs and their accreditation is another area of debate. MOOCs run 
by educational technology companies such as Coursera and EdX provide the option to 
pay for certification. For example, Coursera offers proctored exams for a fee, which will 
earn (if successful) certification. On the other hand, most MOOCs offer badges for 
completion of either the full course or each unit (or week of work, for example 
OLDSMOOC by the Open University). Some MOOC-offering institutions and/or 
instructors provide a Statement of Accomplishment for successful students. However, 
these generally do not carry college credits. Recently, the American Council on 
Education recommended five Coursera courses for college credit (Coursera, 2013) while 
a California bill is seeking credit for students taking faculty approved courses online 
(Levin, 2013). However, given that existing in-person “closed book” examinations are 
subject to cheating (Shimbun, 2011) and that at least one enterprising IT worker in the 
US successfully outsourced his own job to China (Kim, 2013), the challenges of 
validating the assessments of MOOC participants are clear. 


Conclusion 


MOOCs have created wide interest as a change agent in higher education, and the peer- 
reviewed research literature on them is growing but still limited. Many articles 
published to date have discussed empirical evidence from case studies, the influence on 
higher education structure, or educational theory relating to MOOCs. While there is 
research into the learner perspective, neither the creator/facilitator perspective nor the 
technological aspects are being widely researched (or at least such research has not yet 
been published). MOOCs generate a plethora of data in digital form for interested 
researchers. However, this volume has so far limited researchers to analysing only a tiny 
portion of the available data, restricting our understanding of MOOCs. There are further 
interesting research avenues such as cultural tensions within courses and the ethical 
aspects of using data generated by MOOC participants still to be explored. 
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Notes 

1. However, it is worth mentioning that prior to the wide acceptance of a LMS as 
the medium to publish course content, institutions used television broadcasts. 
For example, many people who were not engaged in the Open University 
courses watched various Open University programs telecasted, particularly 
when video recorders became cheaply available. This unregistered informal 
accessing of educational materials reduced when the Open University main 
broadcasts stopped and course materials were delivered primarily through the 
post (on CD and DVD) and later via online delivery. Similar closure happened 
when early adopters of online provision of course content through the open web 
closed off access by placing it behind authentication restrictions in LMS (such 
as Moodle and Blackboard) (Goldberg, 2001). 

2. Automatic fuzziness in search engines may also have reported “Management of 
Change” (MOC) results, particularly given the overlap of “Management of 
(Organizational) Change” MOOC/MOC terminology (Kohil & Kumar, 2011). 

3. The initial full list included: Agency, Apprenticeship, Autonomy, Case study, 
Chaos theory, Cloud computing, Comparative, Connectivism, Actor Network 
Theory, Dangers, Educational theory, Ethics, Framework, Futures, Higher 
education, Information science, Introductory, Learner experience, Learning 
analytics, Opportunities, Pedagogies, Retention, Support, Technology, Trends, 
Opinions, and Reviews. 
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